Papers with scoring metrics
Evaluating adversarial attacks against multiple fact verification systems (D19-1)
Copied to clipboard
| Challenge: | Automated fact verification is progressing due to advances in modeling and availability of large datasets. |
| Approach: | They propose two scoring metrics which take into account the correctness of adversarial instances. |
| Outcome: | The proposed method and paraphrasing method have higher potency and higher resilience than baselines. |
ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments (2022.acl-long)
Copied to clipboard
| Challenge: | Existing automated evaluation systems of chatbots rely on static chat scripts as ground truth, which is hard to obtain. |
| Approach: | They propose an interactive chatbot evaluation framework that allows chatbots to compete with each other like in a sports tournament. |
| Outcome: | The proposed framework can rank chatbots independently from their model architectures and domains . existing evaluation systems rely on static chat scripts as ground truth . |
Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research has evaluated large language models' cultural knowledge and contextual understanding, reducing their effectiveness in multicultural settings. |
| Approach: | They propose a benchmark to evaluate LLMs' cultural understanding with a focus on Korean superstitions. |
| Outcome: | The proposed benchmark assesses multilingual LLMs in Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. |
Regression Aware Inference with LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong results on a range of applications, including regression and scoring tasks. |
| Approach: | They propose alternative inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses. |
| Outcome: | The proposed approach significantly improves over baselines across datasets and models. |